Video feature tagging

ABSTRACT

An activity recording system is provided. The activity recording system includes a three-dimensional camera, a sensor arrangement that is fitted to a subject being recorded, and an activity recording device. The activity recording device receives image information from the three-dimensional camera and sensor arrangement information from the sensor arrangement. Both the image information and the sensor arrangement information include location measurements. The sensor arrangement information is generated by location sensors that are positioned at target features of the subject to be tracked. The sensor arrangement information is a key to the image information that specifies where, in any given image, the target features of the subject lie. Activity data having these characteristics may be applied to solve a variety of system development problems. Such activity data can be used to training machine learning components or test computer vision components for a fraction of the cost of using conventional techniques.

RELATED APPLICATIONS

This application claims benefit under 35 U.S.C. § 120 as a continuationof U.S. patent application Ser. No. 16/185,689, titled “Video FeatureTagging,” filed Nov. 9, 2018, which claims benefit under 35 U.S.C. § 120as a continuation of U.S. patent Ser. No. 14/865,403, titled “VideoFeature Tagging,” filed Sep. 25, 2015 and now U.S. Pat. No. 10,129,530issued Nov. 13, 2018, each of which is hereby incorporated herein byreference in its entirety.

NOTICE OF MATERIAL SUBJECT TO COPYRIGHT PROTECTION

Portions of the material in this patent document are subject tocopyright protection under the copyright laws of the United States andof other countries. The owner of the copyright rights has no objectionto the facsimile reproduction by anyone of the patent document or thepatent disclosure, as it appears in the United States Patent andTrademark Office publicly available file or records, but otherwisereserves all copyright rights whatsoever. The copyright owner does nothereby waive any of its rights to have this patent document maintainedin secrecy, including without limitation its rights pursuant to 37C.F.R. § 1.14.

BACKGROUND

Machine learning programs enable robust computer vision applicationsbecause, once adequately trained, these programs are able to classifyphysical features of a subject despite adverse conditions, such asvariable subject positions, orientations, and lighting. However, to beadequately trained, many machine learning programs require large sets ofvalidated data. For example, in the context of real-time hand tracking,a machine learning algorithm may require hundreds of validated videoframes to be trained adequately. In some instances, to be validated,depth pixels in each of these video frames are manually tagged along theheight and width dimensions to specify the locations of fingertips andother important features captured in the frame. In these instances, themetadata generated by this manual tagging process is used in conjunctionwith the video frames to train a machine learning program to be aclassifier of hand positions, orientations, and translations.Unfortunately, this process of manually tagging video frames islabor-intensive, time-consuming, and prone to error. Furthermore,because classifiers are dependent on the specific camera used duringtraining, new validated data must be created each time a new camera isintroduced. For these reasons, creation of validated data can be abottleneck in the development of many computer vision algorithms.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an activity recording system configured in accordancewith an embodiment of the present disclosure.

FIG. 2 illustrates a sensor arrangement configured in accordance with anembodiment of the present disclosure.

FIG. 3 illustrates an activity recording process in accordance with anembodiment of the present disclosure.

FIG. 4 illustrates a training system configured in accordance with anembodiment of the present disclosure.

FIG. 5 illustrates a testing system configured in accordance with anembodiment of the present disclosure.

FIG. 6 illustrates a computing system configured in accordance with anembodiment of the present disclosure.

FIG. 7 illustrates a mobile computing system configured in accordancewith an embodiment of the present disclosure.

DETAILED DESCRIPTION

Activity recording systems disclosed herein automate the creation ofactivity data that includes images of a subject that are associated with(e.g., tagged with) precise and accurate location and/or orientationmeasurements of the subject. The ability to create three-dimensionalactivity data of this sort differentiates the systems described hereinfrom conventional approaches, such as manual tagging, in which a useridentifies the location of a feature only along the height and widthdimensions. In some embodiments, the activity recording system includesa three-dimensional camera, a sensor arrangement that is fitted to asubject being recorded, and an activity recording device. The activityrecording device receives image information from the three-dimensionalcamera and sensor arrangement information from the sensor arrangement.Both the image information and the sensor arrangement informationinclude location measurements. However, the sensor arrangementinformation is generated by sensors that are positioned on the subjectrelative to target features to be tracked. These target features may bepoints or regions (e.g., a finger or a hand). Each of the sensors in thesensor arrangement may detect, track, and measure location and/ororientation of the target features within a physical space. Thus, sensorarrangement information is a key to the image information that specifieswhere, in any given image, the target features of the subject lie.Activity data having these characteristics may be applied to solve avariety of system development problems. For instance, such activity datacan be used to training machine learning components or test computervision components for a fraction of the cost of using conventionaltechniques.

Still other aspects, embodiments and advantages of these example aspectsand embodiments, are discussed in detail below. Moreover, it is to beunderstood that both the foregoing information and the followingdetailed description are merely illustrative examples of various aspectsand embodiments, and are intended to provide an overview or frameworkfor understanding the nature and character of the claimed aspects andembodiments. References to “an embodiment,” “other embodiments,” “anexample,” “some embodiments,” “some examples,” “an alternateembodiment,” “various embodiments,” “one embodiment,” “at least oneembodiment,” “another embodiment,” “this and other embodiments” or thelike are not necessarily mutually exclusive and are intended to indicatethat a particular feature, structure, or characteristic described inconnection with the embodiment or example may be included in at leastone embodiment or example. The appearances of such terms herein are notnecessarily all referring to the same embodiment or example. Anyembodiment or example disclosed herein may be combined with any otherembodiment or example.

Also, the phraseology and terminology used herein is for the purpose ofdescription and should not be regarded as limiting. Any references toexamples, embodiments, components, elements, or acts of the systems andmethods herein referred to in the singular may also embrace embodimentsincluding a plurality, and any references in plural to any embodiment,component, element or act herein may also embrace embodiments includingonly a singularity. References in the singular or plural form are notintended to limit the presently disclosed systems or methods, theircomponents, acts, or elements. The use herein of “including,”“comprising,” “having,” “containing,” “involving,” and variationsthereof is meant to encompass the items listed thereafter andequivalents thereof as well as additional items. References to “or” maybe construed as inclusive so that any terms described using “or” mayindicate any of a single, more than one, and all of the described terms.In addition, in the event of inconsistent usages of terms between thisdocument and documents incorporated herein by reference, the term usagein the incorporated references is supplementary to that of thisdocument; for irreconcilable inconsistencies, the term usage in thisdocument controls.

General Overview

As previously explained, conventional techniques for tagging videofeatures can be used to create data that describes locations of targetfeatures of a subject, but these techniques have several shortcomings,which are based largely on the fact that the techniques are manual. Twopossible approaches are the use of gloves colored to aid in interpretingfeatures of the hand and motion capture gloves. Unfortunately, theseapproaches also have shortcomings of their own, as will be apparent inlight of this disclosure. For example, colored gloves report globaltranslation error of 5-15 cm, which increases as the hand moves furtheraway from the camera. In addition, colored gloves, which are oftenformed of latex, have an adverse impact on the infrared informationacquired by three-dimensional cameras. Colored gloves also have anadverse impact on the RGB information acquired by cameras since, undernormal operating conditions, hands will not have the complex colorschemes exhibited by the colored gloves. Thus the use of colored glovesis often associated with training errors in machine learning algorithms.Motion capture gloves are also not suitable for video feature taggingbecause these gloves transmit rotational information rather thanpositional information and, therefore, cannot provide positionalinformation regarding some anatomical features of the hand (e.g., thefingertips). In addition, the material of motion capture gloves isopaque, and their use causes training errors in machine learningalgorithms.

Thus, and in accordance with some embodiments of the present disclosure,activity recording devices and systems are provided that generateactivity data in which image data is associated (e.g., tagged) withlocation and/or orientation measurements of a subject's target features.Such activity recording devices and systems differ from conventionalapproaches to video feature tagging at least in that the activityrecording devices and systems automatically create activity data thatassociates the image data with location and/or orientation measurementsof a subject's target features. In one embodiment, the activityrecording devices and systems automatically tag features during thevideo recording process. In this embodiment, the system utilizes acombination of small magnetic sensors, which are fitted to the subjectprior to the recording process, and a three-dimensional camera togenerate accurate tags that remove the need for manual tagging. Theactivity data generated by the system in this embodiment is suitable foruse with any computer vision component. It is appreciated that whilemany of the examples discussed herein focus on hand tracking, thetechniques described herein may be applied to automatically tag featuresof any object.

In some embodiments, the activity recording system includes an activityrecording device in data communication with a three-dimensional cameraand an arrangement of sensors configured to detect location and/ororientation within a physical space. Each of the sensors in thearrangement is positioned to enable recordation of one or more targetfeatures of a subject. Further, in some embodiments, each of the sensorsis attached to the subject using visually unobtrusive components (e.g.,clear tape). Such unobtrusive components minimize any visual artifactsthat may be generated by the sensor arrangement.

In some embodiments, prior to starting a recording session, the activityrecording device calibrates initial information received from the sensorarrangement to initial information received from the camera. Suchcalibration activities are described further below.

In some embodiments, during a recording session, the activity recordingdevice receives image information from the camera and sensor arrangementinformation from the sensor arrangement. In these embodiments, theactivity recording device stores image data based on the imageinformation received and sensor arrangement data based on the sensorarrangement information received in data store. Next, the activityrecording device synchronizes and associates the sensor arrangement datawith the image data. Where the sampling rate of the sensor arrangementand the camera differ, the activity recording device may calculatefabricated sensor readings, as described further below.

After the image data is synchronized and associated with the sensorarrangement data, the resulting activity data may be used by othercomputing systems to test and train computer vision components andmachine learning components. Systems configured to execute theseprocesses are described further below.

System Architecture

FIG. 1 illustrates one example embodiment of an activity recordingsystem 100 that is configured to create synchronized, calibratedactivity data using information received via two distinct channels. Thesystem 100 may be implemented using one or more computing systems, suchas may be fabricated using any of the components of the systemsdescribed below with reference to FIGS. 6 and 7. As shown in FIG. 1, thesystem 100 includes a sensor arrangement 102, a camera 104, and anactivity recording device 106. In some embodiments, the camera 104 is athree-dimensional or depth camera. As shown, the activity recordingdevice 106 is coupled to the sensor arrangement 102 via a dataconnection 132 and coupled to the camera 104 via a data connection 134.As described further below, the activity recording device 106 exchangeslocation and/or orientation information with the sensor arrangement 102via the data connection 132 and exchanges image information with thecamera 104 via the data connection 134.

As shown in FIG. 1, the sensor arrangement 102 includes one or moresensors 108. Each of the sensors 108 is configured to measure locationand/or orientation within a physical space. The one or more sensors 108may include a palm sensor 138. In various embodiments, each of the oneor more sensors 108 is configured to acquire and transmit datadescriptive of physical location and/or orientation. A variety ofsensors may be employed for this purpose, including accelerometers,gyroscopes, magnetometers, and the like. In at least one embodiment,each of the sensors 108 includes an accelerometer and a gyroscope,providing for a 6 degrees of freedom sensor. In some embodiments, eachof the sensors 108 is small enough (e.g., approximately 1 mm) to avoidcreating artifacts in image features captured by the camera 104,regardless of whether those image features are captured via visiblelight, infrared waves, or other electromagnetic radiation. In at leastone embodiment, each of the sensors 108 is included in a commerciallyavailable tracking system, such the trakSTAR™ tracking systemcommercially available from Ascension Technology Corporation ofShelburne, Vt.

As shown in FIG. 1, the sensor arrangement 102 includes 22 sensors 108positioned at various locations about a human hand. As illustrated,these locations are generally positioned near joints to enable to thesensor arrangement 102 to capture location and/or orientationinformation descriptive of the each portion of the hand capable ofsubstantially isolated movement as well as overall movement of the hand.FIG. 2 illustrates another sensor arrangement 200 that may be used inplace of or in addition to the sensor arrangement 102 in someembodiments. As shown, the sensor arrangement 102 includes 6 sensors108. One of the 6 sensors is positioned near the center of the palm ofthe hand and the remaining 5 are positioned near the fingertips toenable the sensor arrangement 102 to capture location and/or orientationinformation descriptive of the fingers relative to the palm of the handas well as overall movement of the hand.

As described further below, in some embodiments, the sensor arrangement102 is housed within a glove. This glove may be fabricated fromtransparent materials, such as translucent fabric or clear vinyl, toprevent the glove from interfering with location information generate bythe camera 104, where, for example, the camera using infrared waves todetect location.

While the sensor arrangements 102 and 200 are configured to capture handlocation and/or orientation, it is appreciated that other sensorarrangements may be configured to capture location and/or orientation ofother anatomical features (e.g., the head, arms, legs, full body, etc.).In addition, it is appreciated that other sensor arrangements may beconfigured to capture locations and/or orientations of other animals,such as common pets, guide dogs, exotic animals, and the like. It isalso appreciated that other sensor arrangements may be configured tocapture locations and/or orientations of other moving objects, such asdrones, cars, boats, and the like. These and other sensor arrangementsmay be used in accord with various embodiments to generate activitydata, such as the activity data 130.

In some embodiments, the activity recording device 106 is implementedusing a computing system, such as one of the computing systems describedfurther below with reference to FIGS. 6 and 7. The activity recordingdevice 106 includes several specialized components that collectivecreate calibrated, synchronized activity data 130 useful for a varietyof applications, some of which are described further below withreference to FIGS. 4 and 5. As shown, the activity recording device 106includes a sensor interface 120, a synchronization engine 122, acalibration engine 136, a camera interface 118, and an activity datastore 124. The activity data store 124 includes the synchronizedactivity data 130 which, in turn, includes image data 126 and sensorarrangement data 128.

As illustrated in FIG. 1, the sensor interface 120 is configured toreceive information descriptive of physical location and/or orientationfrom the sensors 108 via the data connection 132. In some embodiments,the sensor interface 120 is also configured to process the receivedlocation and/or orientation information and store the processedinformation in the activity data store as the sensor arrangement data128. The processing that the sensor interface 120 is configured toperform may include monitoring a system clock implemented by theactivity recording device 106 and storing a current time stamp inassociation with each sensor reading as the reading is received.

As shown in FIG. 1, the camera interface 118 is configured to receiveinformation descriptive of a plurality of images (e.g., video frames)from the camera 104 via the data connection 134. In some embodiments,the camera interface 118 is also configured to process the receivedinformation and store the processed information in the activity datastore as the image data 126. The processing that the camera interface118 is configured to perform may include monitoring a system clockimplemented by the activity recording device 106 and storing a currenttime stamp in association with each image as the image is received.

As illustrated in FIG. 1, the calibration engine 136 is configured toreceive initial image information from the camera interface 118 andinitial sensor arrangement information from the sensor interface 120. Insome embodiments, the calibration engine 136 is also configured togenerate calibration data using the initial image information and theinitial sensor arrangement information. When executing according to itsconfiguration in some embodiments, the calibration engine may generateand store the calibration data as part of the activity data 130 in theactivity data store 124. The calibration data specifies a relationshipbetween location measurements made by the camera 104 and locationmeasurements made by the sensors 108 of sensor arrangement 102. Thecalibration data may be used by components accessing the activity data130 to reconcile location measurements in the sensor arrangement data128 and the image data 126 to a common frame of reference. For example,either or both of the sensor interface 120 and camera interface 118 maybe configured to use the calibration data to reconcile locationinformation in the image information with location information in thesensor arrangement information and store the reconciled locationinformation in the activity data store 124. Alternatively the sensorinterface 120 and/or the camera interface 118 may transmit calibrationdata back to the sensor arrangement 102 and/or the camera 104 so thatboth of these devices transmit location information using a common frameof reference. In another embodiment, other components (e.g., thesynchronization engine 122) that access the activity data 130 may accessthe calibration data to calculate interpretations of the sensorarrangement data 128 that spatially match interpretations of the imagedata 126 where the image data 126 and the sensor arrangement data 128depict the same physical location.

In one embodiment, the calibration engine 136 generates calibration databy comparing a first location of a reference point specified by theinitial image information to a second location of the reference pointspecified by the initial sensor arrangement information. The referencepoint may be, for example, the location of a sensor 108 positioned inthe center of a palm of a hand presented to the camera 104. Where thefirst location does not equal the second location, calibrationinformation is required to provide a common frame of reference betweenthe sensor arrangement 102 and the camera 104. In this instance, thecalibration engine 136 calculates a difference between the firstlocation and the second location and stores the difference ascalibration data for use by other components of the activity recordingsystem 100.

As shown in FIG. 1, the synchronization engine 122 is configured tosynchronize the image data 126 with the sensor arrangement data 128.Such synchronization may be required where, for example, the samplingfrequency of the sensor arrangement 120 differs from the samplingfrequency of the camera 104. When executing according to thisconfiguration in some embodiments, the synchronization engine 122identifies a set of sensor readings that correspond temporally to eachimage of the image data 126. This set of sensor readings may include areading for each sensor 108, some minimum number of sensor readings, orreadings for, at least, some identified subset of the sensors 108. Inone embodiment, the synchronization engine 122 identifies a set ofsensor readings for an image by reading a time stamp associated with theimage and attempting to find one or more matching time stamps in thetime stamps associated with the sensor arrangement data 128. In someembodiments, the synchronization engine 122 identifies a matching timestamp where a time stamp associated with the sensor arrangement data 128is within a configurable threshold value of a time stamp associated animage of the image data 126. Where matching time stamps for one or moresensor readings of the set of sensor readings are not identified, thesynchronization engine 122 may calculate one or more fabricated sensorreadings to complete the set of sensor readings. The synchronizationengine may calculate fabricated sensor readings using any of a varietyof value estimation techniques. For instance, in one embodiment, thesynchronization engine 122 calculates a fabricated sensor reading byinterpolating a value based on two sensor readings associated with timestamps closest to the time stamp associated with the image beingsynchronized. Regardless of the value estimation technique used, thesynchronization engine 122 stores fabricated sensor readings andassociations between each image and its corresponding set of actualand/or fabricated sensor readings in the activity data 130.

Other synchronization techniques may be implemented without departingfrom the scope of the present disclosure. For instance, some embodimentsmay use time stamps that originate from the camera 104 and/or the sensorarrangement 102 in place of the time stamps that originate from theactivity recording device 106 as the basis for the synchronizationprocesses described above. Other embodiments may synchronize the sensorarrangement data 128 directly to the image data 126 by creatingassociations between each received sensor reading and a current image(e.g., the image last received when the sensor reading is received). Inanother embodiment, the camera 104 and/or the sensor arrangement 102 aresynchronized with the activity recording device 106 in hardware (e.g.,via a connection of a microcontroller). Thus, the embodiments disclosedherein are not limited to a particular synchronization data orsynchronization technique.

Methodology

According to some embodiments, an activity recording system (e.g., theactivity recording system 100) executes processes that generate activitydata. FIG. 3 illustrates a data generation process 300 in accord withthese embodiments. As shown in FIG. 3, the detection process 300includes several acts that, in combination, enable the activityrecording system to efficiently generate activity data that can be usedto train classifiers and test other computer vision programs.

In act 302, a subject (e.g., a human hand) is fitted with a sensorarrangement (e.g., the sensor arrangement 102) that is connected to anactivity recording device (e.g., the activity recording device 106). Forexample, in some embodiments, this fitting process may include affixingsensors (e.g., the sensors 108) to a subject using adhesive, such asmedical tape, transparent tape, or the like. In other embodiments, thisfitting process may include placing a transparent glove or some othertransparent garment housing the sensors on the subject. Next, theactivity recording system is powered on and the subject is placed inclear view of a camera (e.g., the camera 104) that is connected to theactivity recording device.

In act 304, the activity recording device calibrates the sensorarrangement to camera by executing a calibration engine (e.g., thecalibration engine 136). In some embodiments, the calibration enginegenerates calibration data that can be used to match a location measuredby a palm sensor (e.g., the palm sensor 138) to a location of the palmsensor measured by the camera. The calibration engine stores thecalibration data within activity data (e.g., the activity data 130). Thecalibration data may be used by components that generate locationmeasurements or access the activity data to spatially reconcile locationmeasurements in sensor arrangement data (e.g., the sensor arrangementdata 128) to location measurements in image data (e.g., the image data126). Examples of the components that generate location measurements oraccess the activity data, according to some embodiments, include thesensor arrangement, the camera, a sensor interface (e.g., the sensorinterface 120), a synchronization engine (e.g., the synchronizationengine 122), a camera interface (e.g., the camera interface 118), andother components.

In act 306, the activity recording device records the subject via afirst channel by receiving, via the camera interface, image informationdepicting the subject, processing the image information as describedabove, and storing the processed image information as the image data. Inact 308, the activity recording device also records the subject via asecond channel by receiving, via the sensor interface, sensorarrangement information depicting the subject, processing the sensorarrangement information as described above, and storing the processedsensor arrangement information as the sensor arrangement data.

In act 310, the activity record synchronizes the first channel to thesecond channel by executing the synchronization engine. As describedabove, in some embodiments, the synchronization engine identifies a setof sensor readings that correspond temporally to each image of the imagedata. Further, the synchronization engine may calculate one or morefabricated sensor readings to include in one or more of these sets ofsensor readings as needed to complete each set of sensor readings. Afterthe act 310, the data generation process ends.

Processes in accord with the data generation process 300 enable activityrecording systems to generate activity data that accurate tracks themovement of a subject. Processes in accord with the data generationprocess 300 can be used to iteratively create, for example, a databaseof gesture data in which each member is a distinct gesture targeted foridentification by a computer vision component. Thus, the activity datacan be advantageously used to configured and/or test by other componentsand systems, such as the systems described further below with referenceto FIGS. 4 and 5.

Example Training System

FIG. 4 illustrates one example of a training system 400 that isconfigured to train a machine learning component to operate as aclassifier using activity data (e.g., the activity data 130). The system400 may be implemented using one or more computing systems, such as maybe fabricated using any of the components of the systems described belowwith reference to FIGS. 6 and 7. As shown in FIG. 4, the system 400includes a machine learning component 402, a training component 404, andan activity data store 406 storing the activity data 130. The machinelearning component 402 maybe any of a variety of machine learningcomponents, such as a support vector machine, a random forest process, aneural network, or the like.

In some embodiments in accord with FIG. 4, the machine learningcomponent 402 is configured to receive and classify image data (e.g.,the image data 126) as being an instance of a class. For example, themachine learning component 402 may be configured to classify each fingerdepicted in the image data as being, for example, a thumb, index finger,middle finger, ring finger, or pinky. In other embodiments, the machinelearning component 402 may be configured to classify the image data intopositive and negative sets answering the query: “Does the gesture in theimage data show the back of the hand?”

To be able to perform accurate classification, the machine learningcomponent 402 must be adequately trained. The training component 404 isconfigured to train the machine learning component 402 using the imagedata and sensor arrangement data (e.g., the sensor arrangement data128). When executing according to this configuration, the machinelearning component 402 executes an iterative training process until someset of convergence criteria are satisfied. In each iteration of thistraining process the machine learning component 402 receives image dataas input and generates output classifying the image data. Also withineach iteration of the training process, the training component receivesthe output and calculates an amount of error in the classification usingthe sensor arrangement data. If the amount of error meets theconvergence criteria, the training component 404 terminates the trainingprocess. If the amount of error does not meet the convergence criteria,the training component 404 adjusts the machine learning component 402 inan attempt to decrease the amount of error in the next iteration andinitiates the next iteration.

Example Testing System

FIG. 5 illustrates one example of a testing system 500 that isconfigured to test operation of a computer vision component 502 usingactivity data (e.g., the activity data 130). The system 500 may beimplemented using one or more computing systems, such as may befabricated using any of the components of the systems described belowwith reference to FIGS. 6 and 7. As shown in FIG. 5, the system 500includes a computer vision component 502, a testing component 504, andan activity data store 506 storing the activity data 130. The computervision component 402 maybe any of a variety of computer visioncomponents that track and processes image data (e.g., the image data126).

In some embodiments in accord with FIG. 5, the computer vision component502 is configured to receive image data and track targeted featurepoints within the image data. For example, the computer vision component502 may be configured to track each finger depicted in the image dataand identify gestures made by the hand as a whole. When executing indebug mode, the computer vision component 402 generates an output filethat includes tracking data. In some embodiments, this tracking dataincludes location and/or orientation measurements of each anatomicalfeature tracked as determined by the computer vision component 402 fromthe image data.

To ensure that the computer vision component 502 is tracking theanatomical features correctly, the testing component 504 is configuredto receive the output file, compare the location measurements in theoutput file to corresponding location measurements in sensor arrangementdata (e.g., the sensor arrangement data 130), and calculate anydifferences between the location measurements in the output file and thelocation measurements in sensor arrangement data. In some embodiments,these differences are summarized to provide an overall assessment of thequality of the computer vision component. For instance, the testingcomponent may be configured to calculate an average difference over allframes in the image data. These differences and summaries may bereported to a test engineer who may analyze them further prior toproviding the test data and comments to developers of the computervision component 502.

Information within the systems disclosed herein may be stored in anylogical and physical construction capable of holding information on acomputer readable medium including, among other structures, linkedlists, file systems, flat files, indexed files, hierarchical databases,relational databases or object oriented databases. The data may bemodeled using unique and foreign key relationships and indices. Theunique and foreign key relationships and indices may be establishedbetween the various fields and tables to ensure both data integrity anddata interchange performance.

Information may flow between the components disclosed herein using avariety of techniques. Such techniques include, for example, passing theinformation over a network using a variety of standards, such asBLUETOOTH, WiFi, USB, TCP/IP or HTTP, passing the information betweenmodules in memory and passing the information by writing to a file,database, data store, or some other non-volatile data storage device. Inaddition, pointers or other references to information may be transmittedand received in place of, in combination with, or in addition to, copiesof the information. Conversely, the information may be exchanged inplace of, in combination with, or in addition to, pointers or otherreferences to the information. Other techniques and protocols forcommunicating information may be used without departing from the scopeof the examples and embodiments disclosed herein.

Each of the processes disclosed herein depict one particular sequence ofacts in a particular example. The acts included in these processes maybe performed by, or using, one or more computer systems speciallyconfigured as discussed herein. Some acts are optional and, as such, maybe omitted in accord with one or more examples. Additionally, the orderof acts can be altered, or other acts can be added, without departingfrom the scope of the systems and methods discussed herein. Furthermore,as discussed above, in at least one example, the acts are performed on aparticular, specially configured machine, namely an activity recordingsystem, a training system, or a testing system configured according tothe examples and embodiments disclosed herein.

Example System

FIG. 6 illustrates a computing system 600 configured in accordance withan embodiment of the present disclosure. In some embodiments, system 600may be a computing system for detecting activity of subjects althoughsystem 600 is not limited to this context. For example, system 600 maybe incorporated into a personal computer (PC), laptop computer,ultra-laptop computer, tablet, touch pad, portable computer, handheldcomputer, palmtop computer, personal digital assistant (PDA), cellulartelephone, combination cellular telephone/PDA, television, smart device(e.g., smart phone, smart tablet or smart television), mobile internetdevice (MID), messaging device, data communication device, set-top box,game console, or other such computing environments capable of performinggraphics rendering operations and displaying content.

In some embodiments, system 600 comprises a platform 602 coupled to adisplay 620. Platform 602 may receive content from a content device suchas content services device(s) 630 or content delivery device(s) 640 orother similar content sources. A navigation controller 650 comprisingone or more navigation features may be used to interact with, forexample, platform 602 and/or display 620, so as to supplementnavigational gesturing by the user. Each of these example components isdescribed in more detail below.

In some embodiments, platform 602 may comprise any combination of achipset 605, processor 610, memory 612, storage 614, graphics subsystem615, applications 616 and/or radio 618. Chipset 605 may provideintercommunication among processor 610, memory 612, storage 614,graphics subsystem 615, applications 616 and/or radio 618. For example,chipset 605 may include a storage adapter (not depicted) capable ofproviding intercommunication with storage 614.

Processor 610 may be implemented, for example, as Complex InstructionSet Computer (CISC) or Reduced Instruction Set Computer (RISC)processors, x86 instruction set compatible processors, multi-core, orany other microprocessor or central processing unit (CPU). In someembodiments, processor 610 may comprise dual-core processor(s),dual-core mobile processor(s), and so forth. Thus processor 610 may beimplemented with a general purpose processor. However, when executing aspecific software process as provided herein, the processor 610 becomesa special purpose processor capable of making specific logic-baseddeterminations based on input data received, and further capable ofproviding one or more outputs that can be used to control or otherwiseinform subsequent processing to be carried out by the processor 610and/or other processors or circuitry with which processor 610 iscommunicatively coupled. Thus, the processor 610 reacts to specificinput stimulus in a specific way and generates a corresponding outputbased on that input stimulus. In this sense, the structure of processor610 according to one embodiment is defined by the processes that itexecutes. In some example cases, the processor 610 proceeds through asequence of logical transitions in which various internal registerstates and/or other bit cell states internal or external to theprocessor 610 may be set to logic high or logic low. This specificsequence of logic transitions is determined by the state of electricalinput signals to the processor 610 and, consequently, a special-purposestructure is effectively assumed by the processor 610 when executingsoftware instructions. Specifically, the instructions anticipate thevarious stimulus to be received and change the implicated memory statesaccordingly. In this way, the processor 610 may generate and store orotherwise provide useful output signals. Thus, it is appreciated thatthe processor 610, during execution of a software process becomes aspecial purpose machine, capable of processing only specific inputsignals and rendering specific output signals based on the one or morelogic decisions performed during execution of each software instruction.As referred to herein, the processor 610 is configured to execute afunction where software is stored in a data store coupled to theprocessor 310 that is configured to cause the processor to proceedthrough a sequence of various logic decisions that result in thefunction being executed.

Memory 612 may be implemented, for instance, as a volatile memory devicesuch as, but not limited to, a Random Access Memory (RAM), DynamicRandom Access Memory (DRAM), or Static RAM (SRAM). Storage 614 may beimplemented, for example, as a non-volatile storage device such as, butnot limited to, a magnetic disk drive, optical disk drive, tape drive,an internal storage device, an attached storage device, flash memory,battery backed-up SDRAM (synchronous DRAM), and/or a network accessiblestorage device. In some embodiments, storage 614 may comprise technologyto increase the storage performance enhanced protection for valuabledigital media when multiple hard drives are included, for example.

Graphics subsystem 615 may perform processing of images such as still orvideo for display or other computations. Graphics subsystem 615 may be agraphics processing unit (GPU) or a visual processing unit (VPU), forexample. An analog or digital interface may be used to communicativelycouple graphics subsystem 615 and display 620. For example, theinterface may be any of a High-Definition Multimedia Interface,DisplayPort, wireless HDMI, and/or wireless HD compliant techniques.Graphics subsystem 615 could be integrated into processor 610 or chipset605. Graphics subsystem 615 could be a stand-alone card communicativelycoupled to chipset 605. The graphics and/or video processing techniquesmay be implemented in various hardware architectures. For example,graphics and/or video functionality may be integrated within a chipset.Alternatively, a discrete graphics and/or video processor may be used.As still another embodiment, the graphics and/or video functions may beimplemented by a general purpose processor, including a multi-coreprocessor. In a further embodiment, the functions may be implemented ina consumer electronics device.

Radio 618 may include one or more radios capable of transmitting andreceiving signals using various suitable wireless communicationstechniques. Such techniques may involve communications across one ormore wireless networks. Exemplary wireless networks include (but are notlimited to) wireless local area networks (WLANs), wireless personal areanetworks (WPANs), wireless metropolitan area network (WMANs), cellularnetworks, and satellite networks. In communicating across such networks,radio 618 may operate in accordance with one or more applicablestandards in any version.

In some embodiments, display 620 may comprise any television or computertype monitor or display. Under the control of one or more softwareapplications 616, platform 602 may display a user interface 622 ondisplay 620.

In some embodiments, content services device(s) 630 may be hosted by anynational, international and/or independent service and thus accessibleto platform 602 via the Internet or other network, for example. Contentservices device(s) 630 may be coupled to platform 602 and/or to display620. Platform 602 and/or content services device(s) 630 may be coupledto a network 660 to communicate (e.g., send and/or receive) mediainformation to and from network 660. Content delivery device(s) 640 alsomay be coupled to platform 602 and/or to display 620. In someembodiments, content services device(s) 630 may comprise a cabletelevision box, personal computer, network, telephone, Internet enableddevices or appliance capable of delivering digital information and/orcontent, and any other similar device capable of unidirectionally orbidirectionally communicating content between content providers andplatform 602 and/display 620, via network 660 or directly. It will beappreciated that the content may be communicated unidirectionally and/orbidirectionally to and from any one of the components in system 600 anda content provider via network 660. Examples of content may include anymedia information including, for example, video, music, graphics, text,medical and gaming content, and so forth.

Content services device(s) 630 receives content such as cable televisionprogramming including media information, digital information, and/orother content. Examples of content providers may include any cable orsatellite television or radio or Internet content providers. Theprovided examples are not meant to limit the present disclosure. In someembodiments, platform 602 may receive control signals from navigationcontroller 650 having one or more navigation features. The navigationfeatures of controller 650 may be used to interact with user interface622, for example. In some embodiments, navigation controller 650 may bea pointing device that may be a computer hardware component(specifically human interface device) that allows a user to inputspatial (e.g., continuous and multi-dimensional) data into a computer.Many systems such as graphical user interfaces (GUI), and televisionsand monitors allow the user to control and provide data to the computeror television using physical gestures, facial expressions, or sounds.

Movements of the navigation features of controller 650 may be echoed ona display (e.g., display 620) by movements of a pointer, cursor, focusring, or other visual indicators displayed on the display. For example,under the control of software applications 616, the navigation featureslocated on navigation controller 650 may be mapped to virtual navigationfeatures displayed on user interface 622, for example. In someembodiments, controller 650 may not be a separate component butintegrated into platform 602 and/or display 620. Embodiments, however,are not limited to the elements or in the context shown or describedherein, as will be appreciated.

In some embodiments, drivers (not shown) may comprise technology toenable users to instantly turn on and off platform 602 like a televisionwith the touch of a button after initial boot-up, when enabled, forexample. Program logic may allow platform 602 to stream content to mediaadaptors or other content services device(s) 630 or content deliverydevice(s) 640 when the platform is turned “off.” In addition, chipset605 may comprise hardware and/or software support for 5.1 surround soundaudio and/or high definition 7.1 surround sound audio, for example.Drivers may include a graphics driver for integrated graphics platforms.In some embodiments, the graphics driver may comprise a peripheralcomponent interconnect (PCI) express graphics card.

In various embodiments, any one or more of the components shown insystem 600 may be integrated. For example, platform 602 and contentservices device(s) 630 may be integrated, or platform 602 and contentdelivery device(s) 640 may be integrated, or platform 602, contentservices device(s) 630, and content delivery device(s) 640 may beintegrated, for example. In various embodiments, platform 602 anddisplay 620 may be an integrated unit. Display 620 and content servicedevice(s) 630 may be integrated, or display 620 and content deliverydevice(s) 640 may be integrated, for example. These examples are notmeant to limit the present disclosure.

In various embodiments, system 600 may be implemented as a wirelesssystem, a wired system, or a combination of both. When implemented as awireless system, system 600 may include components and interfacessuitable for communicating over a wireless shared media, such as one ormore antennas, transmitters, receivers, transceivers, amplifiers,filters, control logic, and so forth. An example of wireless sharedmedia may include portions of a wireless spectrum, such as the RFspectrum and so forth. When implemented as a wired system, system 600may include components and interfaces suitable for communicating overwired communications media, such as input/output (I/O) adapters,physical connectors to connect the I/O adapter with a correspondingwired communications medium, a network interface card (NIC), disccontroller, video controller, audio controller, and so forth. Examplesof wired communications media may include a wire, cable, metal leads,printed circuit board (PCB), backplane, switch fabric, semiconductormaterial, twisted-pair wire, co-axial cable, fiber optics, and so forth.

Platform 602 may establish one or more logical or physical channels tocommunicate information. The information may include media informationand control information. Media information may refer to any datarepresenting content meant for a user. Examples of content may include,for example, data from a voice conversation, videoconference, streamingvideo, email or text messages, voice mail message, alphanumeric symbols,graphics, image, video, text and so forth. Control information may referto any data representing commands, instructions or control words meantfor an automated system. For example, control information may be used toroute media information through a system, or instruct a node to processthe media information in a predetermined manner (e.g., using hardwareassisted for privilege access violation checks as described herein). Theembodiments, however, are not limited to the elements or context shownor described in FIG. 6.

As described above, system 600 may be embodied in varying physicalstyles or form factors. FIG. 7 illustrates embodiments of a small formfactor device 700 in which system 600 may be embodied. In someembodiments, for example, device 700 may be implemented as a mobilecomputing device having wireless capabilities. A mobile computing devicemay refer to any device having a processing system and a mobile powersource or supply, such as one or more batteries, for example.

As previously described, examples of a mobile computing device mayinclude a personal computer (PC), laptop computer, ultra-laptopcomputer, tablet, touch pad, portable computer, handheld computer,palmtop computer, personal digital assistant (PDA), cellular telephone,combination cellular telephone/PDA, television, smart device (e.g.,smart phone, smart tablet or smart television), mobile internet device(MID), messaging device, data communication device, and so forth.

Examples of a mobile computing device also may include computers thatare arranged to be worn by a person, such as a wrist computer, fingercomputer, ring computer, eyeglass computer, belt-clip computer, arm-bandcomputer, shoe computers, clothing computers, and other wearablecomputers. Other examples of mobile computing devices include robots anddrones. In some embodiments, for example, a mobile computing device maybe implemented as a smart phone capable of executing computerapplications, as well as voice communications and/or datacommunications. Although some embodiments may be described with a mobilecomputing device implemented as a smart phone by way of example, it maybe appreciated that other embodiments may be implemented using otherwireless mobile computing devices as well. The embodiments are notlimited in this context.

As shown in FIG. 7, device 700 may comprise a housing 702, a display704, an input/output (I/O) device 706, and an antenna 708. Device 700also may comprise navigation features 712. Display 704 may comprise anysuitable display unit for displaying information appropriate for amobile computing device. I/O device 706 may comprise any suitable I/Odevice for entering information into a mobile computing device. Examplesfor I/O device 706 may include an alphanumeric keyboard, a numerickeypad, a touch pad, input keys, buttons, a camera, switches, rockerswitches, microphones, speakers, voice recognition device and software,and so forth. Information also may be entered into device 700 by way ofmicrophone. Such information may be digitized by a voice recognitiondevice. The embodiments are not limited in this context.

Various embodiments may be implemented using hardware elements, softwareelements, or a combination of both. Examples of hardware elements mayinclude processors, microprocessors, circuits, circuit elements (e.g.,transistors, resistors, capacitors, inductors, and so forth), integratedcircuits, application specific integrated circuits (ASIC), programmablelogic devices (PLD), digital signal processors (DSP), field programmablegate array (FPGA), logic gates, registers, semiconductor device, chips,microchips, chip sets, and so forth. Examples of software may includesoftware components, programs, applications, computer programs,application programs, system programs, machine programs, operatingsystem software, middleware, firmware, software modules, routines,subroutines, functions, methods, procedures, software interfaces,application program interfaces (API), instruction sets, computing code,computer code, code segments, computer code segments, words, values,symbols, or any combination thereof. Whether hardware elements and/orsoftware elements are used may vary from one embodiment to the next inaccordance with any number of factors, such as desired computationalrate, power levels, heat tolerances, processing cycle budget, input datarates, output data rates, memory resources, data bus speeds and otherdesign or performance constraints.

Some embodiments may be implemented, for example, using amachine-readable medium or article which may store an instruction or aset of instructions that, if executed by a machine, may cause themachine to perform a method and/or operations in accordance with anembodiment of the present disclosure. Such a machine may include, forexample, any suitable processing platform, computing platform, computingdevice, processing device, computing system, processing system,computer, processor, or the like, and may be implemented using anysuitable combination of hardware and software. The machine-readablemedium or article may include, for example, any suitable type of memoryunit, memory device, memory article, memory medium, storage device,storage article, storage medium and/or storage unit, for example,memory, removable or non-removable media, erasable or non-erasablemedia, writeable or re-writeable media, digital or analog media, harddisk, floppy disk, Compact Disk Read Only Memory (CD-ROM), Compact DiskRecordable (CD-R), Compact Disk Rewriteable (CD-RW), optical disk,magnetic media, magneto-optical media, removable memory cards or disks,various types of Digital Versatile Disk (DVD), a tape, a cassette, orthe like. The instructions may include any suitable type of executablecode implemented using any suitable high-level, low-level,object-oriented, visual, compiled and/or interpreted programminglanguage.

FURTHER EXAMPLE EMBODIMENTS

The following examples pertain to further embodiments, from whichnumerous permutations and configurations will be apparent.

Example 1 is an activity recording system comprising an activityrecording device including: a memory; and at least one processor coupledto the memory and configured to receive image information indicating atleast one first three-dimensional measurement of at least one locationof a subject; receive sensor arrangement information indicating at leastone second three-dimensional measurement of the at least one location ofthe subject; and store at least one association between the at least onefirst three-dimensional measurement and the at least one secondthree-dimensional measurement.

Example 2 includes the subject matter of Example 1, wherein the imageinformation depicts a human hand including one or more fingers and theat least one location includes one or more locations of the one or morefingers.

Example 3 includes the subject matter of either Example 1 or Example 2,wherein the at least one processor is further configured to store, inthe memory, image data based on the image information; store, in thememory, sensor arrangement data based on the sensor arrangementinformation; and synchronize the sensor arrangement data to the imagedata.

Example 4 includes the subject matter of Example 3, wherein the at leastone processor is further configured to synchronize the sensorarrangement data to the image data at least in part by being configuredto generate one or more fabricated sensor readings.

Example 5 includes the subject matter of any of the preceding Examples,wherein the at least one processor is further configured to receiveinitial image information; receive initial sensor arrangementinformation; and calibrate the initial sensor arrangement information tothe initial sensor arrangement information.

Example 6 includes the subject matter of any of the preceding Examples,further comprising a three-dimensional camera in data communication withthe activity recording system.

Example 7 includes the subject matter of any of the preceding Examples,further comprising a sensor arrangement in data communication with theactivity recording system.

Example 8 includes the subject matter of Example 7, wherein the sensorarrangement includes 22 sensors.

Example 9 includes the subject matter of Example 7, wherein the sensorarrangement includes 6 sensors.

Example 10 includes the subject matter of any of Examples 7 throughExample 9, wherein the sensor arrangement is included in a transparentgarment.

Example 11 includes the subject matter of Example 10, wherein thetransparent garment includes a glove.

Example 12 is a method of recording activity using a device, the methodcomprising: receiving image information indicating at least one firstthree-dimensional measurement of at least one location of a subject;receiving sensor arrangement information indicating at least one secondthree-dimensional measurement of the at least one location of thesubject; and storing at least one association between the at least onefirst three-dimensional measurement and the at least one secondthree-dimensional measurement.

Example 13 includes the subject matter of Example 12, wherein receivingthe image information includes receiving image information depicting ahuman hand including one or more fingers and indicating one or morelocations of the one or more fingers.

Example 14 includes the subject matter of either Example 12 or Example13, further comprising storing, in the memory, image data based on theimage information; storing, in the memory, sensor arrangement data basedon the sensor arrangement information; and synchronizing the sensorarrangement data to the image data.

Example 15 includes the subject matter of Example 14, whereinsynchronizing the sensor arrangement data to the image data includes togenerating one or more fabricated sensor readings.

Example 16 includes the subject matter of either Example 14 or Example15, further comprising training a machine learning component with theimage data, the sensor arrangement data, and the at least oneassociation.

Example 17 includes the subject matter of any of Examples 14 through 16,further comprising testing a computer vision component with the imagedata, the sensor arrangement data, and the at least one association.

Example 18 includes the subject matter of any of Examples 12 through 17,further comprising receiving initial image information; receivinginitial sensor arrangement information; and calibrating the initialsensor arrangement information to the initial sensor arrangementinformation.

Example 19 includes the subject matter of any of Examples 12 through 18,wherein receiving the image information includes receiving imageinformation from a three-dimensional camera.

Example 20 includes the subject matter of any of Examples 12 through 19,wherein receiving the sensor arrangement information includes receivingsensor arrangement information from 22 sensors.

Example 21 includes the subject matter of any of Examples 12 through 19,wherein receiving the sensor arrangement information includes receivingsensor arrangement information from 6 sensors.

Example 22 is a non-transient computer program product encoded withinstructions that when executed by one or more processors cause aprocess for recording activity to be carried out, the processcomprising: receiving image information indicating at least one firstthree-dimensional measurement of at least one location of a subject;receiving sensor arrangement information indicating at least one secondthree-dimensional measurement of the at least one location of thesubject; and storing at least one association between the at least onefirst three-dimensional measurement and the at least one secondthree-dimensional measurement.

Example 23 includes the subject matter of Example 22, wherein receivingthe image information includes receiving image information depicting ahuman hand including one or more fingers and indicating one or morelocations of the one or more fingers.

Example 24 includes the subject matter of either Example 22 or Example23, the process further comprising storing, in the memory, image databased on the image information; storing, in the memory, sensorarrangement data based on the sensor arrangement information; andsynchronizing the sensor arrangement data to the image data.

Example 25 includes the subject matter of Example 24, whereinsynchronizing the sensor arrangement data to the image data includes togenerating one or more fabricated sensor readings.

Example 26 includes the subject matter of either Example 24 or Example25, the process further comprising training a machine learning componentwith the image data, the sensor arrangement data, and the at least oneassociation.

Example 27 includes the subject matter of any of Examples 24 through 26,the process further comprising testing a computer vision component withthe image data, the sensor arrangement data, and the at least oneassociation.

Example 28 includes the subject matter of any of Examples 22 through 27,the process further comprising receiving initial image information;receiving initial sensor arrangement information; and calibrating theinitial sensor arrangement information to the initial sensor arrangementinformation.

Example 29 includes the subject matter of any of Examples 22 through 28,wherein receiving the image information includes receiving imageinformation from a three-dimensional camera.

Example 30 includes the subject matter of any of Examples 22 through 29,wherein receiving the sensor arrangement information includes receivingsensor arrangement information from 22 sensors.

Example 31 includes the subject matter of any of Examples 22 through 29,wherein receiving the sensor arrangement information includes receivingsensor arrangement information from 6 sensors.

The foregoing description of example embodiments has been presented forthe purposes of illustration and description. It is not intended to beexhaustive or to limit the present disclosure to the precise formsdisclosed. Many modifications and variations are possible in light ofthis disclosure. It is intended that the scope of the present disclosurebe limited not by this detailed description, but rather by the claimsappended hereto. Future filed applications claiming priority to thisapplication may claim the disclosed subject matter in a differentmanner, and may generally include any set of one or more limitations asvariously disclosed or otherwise demonstrated herein.

What is claimed is:
 1. An activity recording system comprising: amemory; and at least one processor coupled to the memory and configuredto: receive image information indicating at least one firstthree-dimensional measurement of at least one location of at least onefinger of a subject, the image information captured by athree-dimensional camera; receive sensor arrangement informationindicating at least one second three-dimensional measurement of the atleast one location of the at least one finger of the subject, the sensorarrangement information captured by a sensor arrangement coupled to theat least one location of the subject; and generate activity data basedon the at least one first three-dimensional measurement with the atleast one second three-dimensional measurement.
 2. The activityrecording system of claim 1, further comprising the three-dimensionalcamera and the sensor arrangement.
 3. The activity recording system ofclaim 1, wherein the at least one processor is further configured to oneor both of: store the activity data within a gesture database; and traina machine learning component to operate as a classifier using theactivity data.
 4. The activity recording system of claim 3, wherein theclassifier is configured to classify one or more fingers depicted inimage data corresponding to the activity data as being one or more of athumb, an index finger, a middle finger, a ring finger, and a pinky. 5.The activity recording system of claim 3, wherein the classifier isconfigured to classify activity data into a first set that correspondsto image data depicting a back of a hand and a second set thatcorresponds to image data not depicting the back of the hand.
 6. Theactivity recording system of claim 3, wherein to train the machinelearning component comprises to: provide image data as input to themachine learning component; receive output classifying the image datafrom the machine learning component; calculate an amount of error in theoutput using activity data corresponding to the image data; and adjustthe machine learning component where the amount of error does not meetconvergence criteria.
 7. The activity recording system of claim 3,wherein the machine learning component comprises one, two, or all threeof a support vector machine, a random forest process, and an artificialneural network.
 8. The activity recording system of claim 1, furthercomprising a glove housing the sensor arrangement.
 9. The activityrecording system of claim 8, wherein the glove is transparent.
 10. Amethod of recording activity using a device, the method comprising:receiving image information indicating at least one firstthree-dimensional measurement of at least one location of at least onefinger of a subject, the image information captured by athree-dimensional camera; receiving sensor arrangement informationindicating at least one second three-dimensional measurement of the atleast one location of the at least one finger of the subject, the sensorarrangement information captured by a sensor arrangement coupled to theat least one location of the subject; and generating activity data basedon the at least one first three-dimensional measurement with the atleast one second three-dimensional measurement.
 11. The method of claim10, further comprising one or both of: storing the activity data withina gesture database; and training a machine learning component to operateas a classifier using the activity data.
 12. The method of claim 11,wherein training the machine learning component comprises training themachine learning component to classify one or more fingers depicted inimage data corresponding to the activity data as being one or more of athumb, an index finger, a middle finger, a ring finger, and a pinky. 13.The method of claim 11, wherein training the machine learning componentcomprises training the machine learning component to classify activitydata into a first set that corresponds to image data depicting a back ofa hand and a second set that corresponds to image data not depicting theback of the hand.
 14. The method of claim 11, wherein training themachine learning component comprises: providing image data as input tothe machine learning component; receiving output classifying the imagedata from the machine learning component; calculating an amount of errorin the output using activity data corresponding to the image data; andadjusting the machine learning component where the amount of error doesnot meet convergence criteria.
 15. A non-transient computer programproduct encoded with instructions that when executed by one or moreprocessors cause a process for recording activity to be carried out, theprocess comprising: receiving image information indicating at least onefirst three-dimensional measurement of at least one location of at leastone finger of a subject, the image information captured by athree-dimensional camera; receiving sensor arrangement informationindicating at least one second three-dimensional measurement of the atleast one location of the at least one finger of the subject, the sensorarrangement information captured by a sensor arrangement coupled to theat least one location of the subject; and generating activity data basedon the at least one first three-dimensional measurement with the atleast one second three-dimensional measurement.
 16. The computer programproduct of claim 15, the process further comprising on or both of:storing the activity data within a gesture database; and training amachine learning component to operate as a classifier using the activitydata.
 17. The computer program product of claim 16, wherein the machinelearning component is configured to classify one or more fingersdepicted in image data corresponding to the activity data as being oneor more of a thumb, an index finger, a middle finger, a ring finger, anda pinky.
 18. The computer program product of claim 16, wherein themachine learning component is configured to classify activity data intoa first set that corresponds to image data depicting a back of a handand a second set that corresponds to image data not depicting the backof the hand.
 19. The computer program product of claim 16, whereintraining the machine learning component comprises: providing image dataas input to the machine learning component; receiving output classifyingthe image data from the machine learning component; calculating anamount of error in the output using activity data corresponding to theimage data; and adjusting the machine learning component where theamount of error does not meet convergence criteria.
 20. The computerprogram product of claim 16, wherein the machine learning componentcomprises one, two, or all three of a support vector machine, a randomforest process, and an artificial neural network.